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ABSTRACT 

The  focus  of  this  work  is  to  understand  impactful  aspects  of  team  functioning  as 
they  solve  complex  problems,  and  propose  the  means  to  improve  the  performance  of 
teams,  under  changing  or  adversarial  conditions. 

By  analyzing  the  characteristic  differences  between  high  performing  and  low 
perfonning  teams,  we  identified  behavior  that  characterized  and  differentiated  the 
problem  solving  approach  of  the  teams  especially  when  the  problem  itself  changes  during 
solving.  We  then  derived  a  simulated  annealing-based  algorithm  that  mimicked  human 
team  problem  solving  under  these  conditions;  aspects  of  the  algorithm  also  performed 
well  as  an  optimization  algorithm  when  solving  a  variety  of  hard  numerical  problems. 

Identification  of  the  impact  of  team  structure  on  problem  solving  behavior  under 
changing  conditions  indicates  that  heterogeneous-grouped  teams  outperfonn 
homogeneous  or  non-grouped  teams.  Yet  solving  multiple  problems  sequentially  is  not  as 
effective  as  breaking  problems  into  smaller  intennixed  chunks,  breaking  fixation. 

Finally,  a  meta-analysis  on  the  design  literature  where  the  impact  of  examples  on 
problem  solving  was  studied  within  the  design  process  identified  under  what  conditions 
examples  will  induce  fixation  or  alternatively,  inspiration. 


STUDY  1:  Effect  of  Team  Structure  on  Problem  Solving  Under  Change  (Sio, 
Kotovsky  and  Cagan,  2014b) 

In  this  study,  we  conducted  an  experiment  and  computational  simulations  to 
look  at  how  team  structure  affects  the  information  processing  within  a  diverse  team, 
and  how  this  affects  team  performance  and  resilience  to  change. 


Experiment:  120  participants  were  assigned  in  teams  of  four  to  make 
decisions  in  which  they  were  given  performance  data  and  had  to  evaluate  some 
companies  and  pick  the  best  one.  Two  members  of  each  team  were  told  to  evaluate 
the  companies  based  on  company  performance,  and  the  other  two  were  told  to  focus 
on  industry  trends.  Teams  were  in  different  subgrouping  structures:  homogeneous, 
heterogeneous,  and  no-subgrouping.  In  a  homogeneous  team,  members  studying  the 
same  domain  of  information  formed  a  subgroup  to  evaluate  the  companies  together. 
The  entire  team  then  had  a  discussion  and  made  a  decision  on  the  best  company. 
The  heterogeneous  teams  followed  a  similar  procedure  except  that  each  subgroup 
was  mixed,  including  one  member  from  each  orientation.  Members  in  the  no¬ 
subgrouping  teams  evaluated  the  companies  alone,  followed  by  a  team  discussion. 
There  were  20  trials.  Industry  trends  were  driving  the  correct  decision  for  the  first 
10  trials.  From  trial  11  to  20,  a  significant  change  was  introduced;  company 
performance  became  the  most  important  factor.  Teams  were  not  informed  of  the 
change,  only  whether  their  choice  was  correct. 

The  expectation  was  that  members  in  the  homogeneous  subgroup  would 
validate  each  other’s  perspective  because  of  the  high  functional  similarity  between 
them.  Thus,  they  should  be  fixated  on  their  own  preference,  making  them  less  able 
to  combine  different  perspectives  in  decision-making  and  adapt  to  the  change.  The 
20  trials  were  divided  into  4  equal  blocks,  representing  the  early  and  late  trials, 
presented  before  and  after  the  change,  for  comparison  purposes.  As  predicted,  the 
homogeneous  team  performed  significantly  worse  than  the  no-subgrouping  and 
heterogeneous  teams  in  the  1st  block,  suggesting  that  the  homogeneous  team  was 
slower  in  learning  the  task.  Further,  all  teams  were  impaired  by  the  change  to  the 
same  degree,  indicated  by  the  non-significant  team  performance  difference  in  the 
3rd  block.  The  performance  difference  re-emerged  in  the  4th  block  where  the 
homogeneous  team  performed  worse  than  the  other  two  teams,  suggesting  that  the 
homogeneous  team  was  less  able  to  adapt  to  the  change  (see  Figure  1).  This  can  be 
viewed  as  a  form  of  fixation. 
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FIGURE  1.  Team  performance  by  block  and  team  structure. 


Computational  Modeling:  Three  3-layer  (an  input,  a  hidden  layer,  and  an 
output)  neural  network  models  (see  Figure  2)  were  built  to  simulate  the  results  of 
each  condition  of  the  experiment.  Each  input  node  received  the  evaluation  by  one 
member  as  input,  and  forwarded  it  to  the  hidden  nodes.  The  three  models  differed 
in  the  interconnectivity  between  the  input  and  hidden  layers,  representing  the 
homogeneous,  heterogeneous,  and  no-subgrouping  structures.  To  simulate  the 
communication  patterns  of  the  team  discussion,  the  hidden  nodes  were  fully- 
connected  with  each  other  allowing  them  to  inhibit  or  excite  each  other. 

Models  were  first  trained  on  the  pre-change  training  set  for  200  cycles,  and 
then  on  the  post-change  set  for  another  200  cycles.  Figure  3  presents  the 
performance  (number  of  errors)  of  the  models  when  learning  the  pre-  and  post¬ 
change  trials.  At  the  beginning,  the  heterogeneous  and  no-subgrouping  models 
demonstrated  a  steeper  decline  in  errors  than  the  homogeneous  model.  The 
differences  then  diminished  over  time.  All  models  were  impaired  by  the  criteria 
change  in  the  training  set  of  data,  indicated  by  the  significant  increase  in  error 
immediately  after  the  change.  All  models  demonstrated  a  decrease  in  error  after  the 
first  few  cycles  of  training.  But  the  homogeneous  model  ceased  to  improve  after 
that.  Only  the  heterogeneous  and  no-subgrouping  models  continued  to  improve, 
showing  a  rapid  and  effective  response  to  the  changed  conditions. 

The  results  of  these  experimental  and  computational  studies  demonstrate  a 
sizable  impact  of  team  communication  structure  and  representational  convergence 
on  a  team’s  ability  to  avoid  fixation  and  respond  to  changed  situations. 

This  work  is  accepted  for  publication  in  CogSci  2014  (Sio,  et  al.,  2014). 
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FIGURE  2.  The  architecture  of  the  models. 
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STUDY  2:  Strategy  Characteristics  of  High  vs.  Low  Performing  Teams  in 
responding  to  sudden  changes  in  problem  conditions.  (McComb,  Cagan  and 
Kotovsky,  2014  &  2015a), 

Differences  in  problem-solving  processes  could  result  from  the  inherent 
variability  of  the  individuals  composing  the  team,  where  individual-level  domain 
experience  could  lead  some  individuals  to  perform  more  like  experts  than  others, 
inducing  team-level  differences.  This  led  us  to  hypothesize  that  teams  that  excelled  in 
responding  to  change  would  display  underlying  problem-solving  processes  that  differed 
from  teams  that  responded  slowly  or  poorly. 

We  explored  this  hypothesis  through  a  cognitive  study  that  tasked  small  teams  of 
undergraduate  engineering  students  with  the  design  of  a  truss  structure.  Midway  through 
the  study,  a  fundamental  aspect  of  the  original  design  problem  was  changed.  Shortly 
thereafter,  yet  another  modification  was  made  to  the  original  design  problem.  The  second 
change  was  intended  to  amplify  the  differences  between  high-  and  low-perfonning  teams 


by  placing  an  emphasis  on  ongoing  flexibility  and  responsiveness.  During  solving,  a 
complete  record  of  the  design  team’s  efforts  was  collected  through  a  computer  interface, 
designed  to  allow  problem-solving  strategies  to  be  fully  reconstructed  for  analysis.  Upon 
reconstruction,  individual  designs  were  analyzed  in  terms  of  quality.  We  quantified 
quality  using  the  strength-to-weight  ratio  (SWR),  nonnalized  according  to  the  objectives 
given  to  the  teams.  The  level  of  divergence  exhibited  by  teams  was  also  analyzed.  This 
was  quantified  by  computing  the  average  pairwise  distance  between  designs  being 
pursued  by  all  pairs  of  members  of  a  team  during  the  entire  solution  process. 

To  compare  the  high-  and  low-performing  teams,  teams  were  first  sorted 
according  to  average  perfonnance.  The  five  highest  and  five  lowest  performing  teams 
were  then  selected  for  further  analysis.  The  SWR  of  the  best  designs  pursued  by  these 
teams  are  shown  in  Figure  4.  There  was  a  considerable  difference  in  the  SWR  of  the 
problem  design  solutions  produced  for  the  first  problem  statement  and  third  (second 
change)  problem  statement,  with  a  smaller  difference  exhibited  for  the  second  (first 
change)  one.  Further,  high-performing  teams  also  showed  a  pattern  of  divergence  that 
was  quite  different  from  the  low-performing  teams.  Figure  5  shows  the  average  intra¬ 
team  pairwise  distance  through  the  study.  In  general,  low-performing  teams  continued  to 
diverge,  moving  far  from  their  previous  solution.  On  the  other  hand,  high-performing 
teams  explored  more  focused  portions  of  the  design  space.  A  more  detailed  examination 
of  designs  produced  by  the  high-  and  low-performing  teams  indicated  that  high- 
performing  teams  tended  to  pursue  simple  designs,  while  low-performing  teams  pursued 
more  complex  but  less  efficacious  designs  (examples  are  provided  in  Figure  6). 

The  differences  between  high-  and  low-performing  teams  can  be  explained  in  part 
through  cognitive  load  theory.  Cognitive  load  theory  states  that  the  cognitive  load 
associated  with  a  task  is  composed  of  intrinsic  load  (difficulty  of  the  task  itself), 
extraneous  load  (additional  load  related  to  method  of  delivery),  and  germane  load  (load 
devoted  to  understanding  infonnation)  (Kirschner,  2002).  Through  producing  simple 
designs,  the  high-perfonning  teams  placed  little  extraneous  cognitive  load  on  themselves, 
which  enabled  them  to  better  understand  the  design  space.  This  augmented  their  ability  to 
respond  quickly  to  changes.  Members  in  low-perfonning  teams  tended  towards  more 
complex  designs,  which  inhibited  this  ability. 


The  concept  of  expertise  can  also  be  utilized  to  interpret  the  results.  It  has  been 
demonstrated  that  problem-domain  experts  can  quickly  and  accurately  classify  problems, 
and  begin  moving  more  or  less  directly  towards  a  solution  (Chi,  Feltovich,  &  Glaser, 
1981).  Specifically  in  the  context  of  design,  expert  designers  tend  to  quickly  commit  to  a 
single  solution  concept,  rather  than  exploring  a  variety  of  alternatives  (Cross,  2004). 
While  true  expertise  is  generally  the  result  of  years  of  effort,  individuals  in  high- 
performing  teams  still  behaved  in  some  ways  like  experts,  quickly  selecting  a  good 
direction  in  which  to  search.  In  addition,  they  exhibited  a  low  level  of  divergence,  similar 
to  the  balanced  search  strategies  observed  in  expert  designers  by  Fricke  (1996). 

The  general  strategy  that  the  high-perfonning  teams  used  to  solve  the  problem  has 
been  identified  (a  combination  of  exploring  simple  solutions,  and  limiting  divergent 
search).  For  some  problem-solving  tasks,  it  may  be  beneficial  to  teach  this  strategy  to 
teams  in  order  to  increase  the  likelihood  of  expert-like  behavior.  This  work  is  accepted 
for  publication  in  the  ASME  Design  Theory  and  Methodology  Conference  (part  of 
IDETC)  in  2014  (McComb  et  al,  2014). 


FIGURE  4.  SWR  (Strength-to-weight  ratio)  of  best  designs  for  high-  and  low-performing 
teams  (error  bars  show  ±1  S.E) 


FIGURE  5.  Average  pairwise  distance  of  high-  and  low-performing  teams  (Error  bars 
show±l  S.E) 
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FIGURE  6.  Representative  solutions  to  the  second  problem  statement  for  a  high- 
performing  team  (left)  and  a  low-performing  team  (right) 


STUDY  3:  Computational  modeling  of  teams  that  exhibit  high  performing 
characteristics  under  dynamic  conditions  (McComb,  Cagan  and  Kotovsky,  2015b  & 
2015e) 

This  work  focuses  on  developing  a  better  understanding  of  team-based  design 
through  computationally  simulating  the  team  design  process  by  introducing  a 
Cognitively-Inspired  Simulated  Annealing  Teams  (CISAT)  modeling  framework.  The 
CISAT  framework  makes  use  of  simulated  annealing  constructs  to  model  several 
characteristics  of  individuals  in  small  design  teams.  It  differs  from  other  simulation 
models  because  it  strikes  a  balance  between  model  simplicity  and  direct  applicability, 
offering  a  succinct  modeling  framework  that  can  be  used  to  directly  solve  engineering 
design  problems.  Further  the  framework  enables  an  analysis  of  which  attributes  of  a 
solution  strategy  most  impact  the  solution  outcome. 

The  CISAT  framework  models  8  characteristics  supported  by  the  literature, 

including  the  results  of  Study  2  above,  that  contribute  to  a  description  of  how  individuals 
solve  problems,  both  independently  and  as  part  of  a  team.  These  characteristics  are  listed 
briefly  below,  and  explained  in  greater  detail  in  subsequent  sections: 

1 .  Multi-agency:  A  team  is  a  collection  of  individuals  with  a  common  goal. 

2.  Organic  interaction  timing:  Interaction  within  teams  occurs  at  irregular 


intervals. 


3.  Quality-informed  solution  sharing:  Members  of  a  team  tend  to  focus  on  the 
most  promising  alternatives,  but  don’t  do  so  myopically. 

4.  Quality  bias  reduction:  Individuals  in  a  team  develop  multiple  solution  concepts 
to  avoid  premature  convergence. 

5.  Self-bias:  Designers  tend  to  be  biased  in  favor  of  their  own  designs. 

6.  Operational  learning:  Individuals  leam  strategies  over  the  course  of  solving. 

7.  Locally  Sensitive  Search:  Designers  select  from  a  range  of  breadth-  and  depth- 
first  search  strategies  as  they  explore  the  design  space. 

8.  Satisficing:  Individuals  only  search  the  solution  space  until  a  solution  is  found 
that  satisfies  relevant  targets. 

The  modeling  framework  is  based  upon  collaboration  between  multiple  software 
agents.  For  CISAT  agents,  the  environment  is  the  problem  space,  and  they  sense  it  by 
evaluating  potential  solutions.  Agents  then  respond  by  creating,  sharing,  and  refining 
solution  concepts.  Within  CISAT,  every  human  designer  is  modeled  by  exactly  one 
agent.  These  agents  share  a  common  goal  (the  minimization  of  an  objective  function) 
making  them  a  suitable  proxy  for  members  of  a  team  (Figure  7).  The  agents  search  based 
on  a  simulated  annealing  algorithm,  modified  by  the  above  characteristics. 


FIGURE  7.  Conceptual  flowchart  for  CISAT  framework 


(a)  SWR  of  best  design 
(cognitive  study  results) 


(b)  SWR  of  best  design 
(CISAT  simulation  results) 


Time  (Mnutnl  Time 


(c)  Average  pairwise  distance 
(Cognitive  study  results) 


(d)  Average  pairwise  distance 
(CISAT  simulation  results) 


FIGURE  8.  Comparison  between  cognitive  study  results  and  CISAT  simulation 

results 


The  CISAT  framework  reproduces  several  of  the  main  trends  that  are  apparent  in 
the  original  results  from  the  cognitive  study.  Among  other  similarities,  the  high- 
performing  human  teams  showed  an  early  divergent  period,  followed  by  a  pattern  of 
fairly  constant  average  pairwise  distance  (see  Figure  8c).  This  is  echoed  in  the  CISAT 
simulation  (see  Figure  8d).  The  low-perfonning  human  teams  show  higher  average 
pairwise  distance,  and  a  period  of  divergence  near  the  end  of  the  study.  This  behavior  is 
also  evidenced  in  the  CISAT  simulation.  As  seen  in  Figure  8  a&b,  even  the  overall  trends 
for  the  calculation  of  the  design  goals  (SWR  -  strength  to  weight  ratio)  is  consistent 
although  the  actual  values  are  slightly  lower  in  the  simulation. 


CISAT  was  used  to  explore  the  particular  characteristics  that  were  most  beneficial 
to  teams  in  solving  this  task.  This  analysis  indicated  that  proper  interaction  (specifically 
self-bias  and  organic  interaction  timing)  was  crucial  to  enabling  team  success  in  the  truss 
design  task.  Further  analysis  revealed  the  importance  of  flexible  design  methods  that 
allow  for  sufficient,  but  not  excessive,  divergent  search. 

STUDY  4:  Using  CISAT  as  the  basis  for  effective  numerical  optimization 
(McComb,  Cagan  and  Kotovsky,  2015c  &  2015d) 

This  CISAT  work  provides  the  basis  for  a  novel  and  effective  optimization 

method.  Called  the  Heterogeneous  Simulated  Annealing  Team  (HSAT)  algorithm,  this 
multi-agent  simulated  annealing  algorithm  includes  multiple  agents  that  each  provide 
their  own  adaptive  annealing  schedule,  allowing  agents  to  develop  heterogeneous  search 
strategies.  Such  diversity,  appropriately  delimited,  is  a  natural  part  of  engineering  design, 
and  also  boosts  perfonnance  in  other  multi-agent  design  algorithms.  Further,  interaction 
between  agents  in  HSAT  is  structured  to  mimic  interaction  between  members  of  a  design 
team  as  derived  in  Study  3  above.  The  perfonnance  of  this  new  algorithm  is  compared  to 
several  other  simulated  annealing  based  algorithms,  as  well  as  a  random  search 
algorithm,  a  gradient-based  algorithm,  and  a  genetic  algorithm.  Although  the  HSAT 
algorithm  tends  to  display  slow  initial  improvement,  it  provides  terminal  solutions  that 
are  better  on  average  than  other  algorithms  explored  in  this  work. 

A  general  flowchart  is  provided  in  Figure  9  to  summarize  the  HSAT  algorithm. 


Cumulative  Distribution  Function 


independently  from  other  agents. 


FIGURE  9.  Flowchart  for  the  HSAT  algorithm. 


(a)  Cumulative  distribution  of  terminal  solution  (b)  Best  solution  found  over  time. 
FIGURE  10.  Comparison  of  optimization  results  for  Ackley  function. 


The  algorithm  has  been  applied  to  several  objective  functions  of  multiple  minima. 
The  results  in  comparison  to  a  variety  of  other  optimization  methods  are  shown  in  Figure 


10  for  the  Ackley  function.  (In  Figure  10a  higher  values  are  better  and  in  Figure  10b 
lower  values  are  better.)  Most  impressive  in  this  instance  is  HSAT’s  ability  to  go  beyond 
the  other  algorithms  to  find  a  much  better  solution. 


STUDY  5:  Overcoming  fixation  in  misleading  situations  through 
interruption  (Sio,  Kotovsky  and  Cagan,  2014a) 

Many  studies  investigating  the  unconscious  processing  that  lead  to  insight  assume 
that  incubation  and  subsequent  insight  experiences  begin  with  an  impasse.  Our  study 
challenges  this  assumption.  Participants  were  asked  to  solve  a  set  of  remote  associate 
tasks  in  two  test  sessions,  with  or  without  incubation  in  between.  Half  of  the  tasks  were 
designed  to  activate  a  misleading  solution  attempt  whereas  the  others  were  neutral.  The 
degree  of  impasse  was  manipulated  by  presenting  the  tasks  for  one  long  time  each  or 
briefly  for  multiple  times  (with  total  time  kept  equivalent)  intermixing  the  two  tasks  with 
each  other.  The  multiple-presentation  group  should  be  less  likely  to  reach  an  impasse 
because  they  could  only  approach  the  problem  briefly  each  time.  However,  this  group 
perfonned  as  well  as  the  single-presentation  group  on  the  neutral  tasks  and  solved  more 
misleading  tasks  than  the  single-presentation  group.  Both  groups  benefited  from 
incubation.  We  conclude  that  unconscious  insight-related  processes  are  not  impasse- 
driven  and  prolonged  conscious  work  may  induce  fixation. 

In  this  study,  participants  were  asked  to  solve  neutral  and  misleading  RAT 
problems  presented  in  a  massed  (30sxl)  or  spaced  (10sx3)  manner,  with  or  without 
incubation.  In  the  first  session,  the  10sx3  and  30sxl  groups  solved  the  same  amount  of 
neutral  RAT  problems,  and  the  10sx3  group  solved  more  misleading  RAT  problems  than 
did  the  30sxl  group.  Both  groups  showed  a  significant  and  comparable  incubation  effect 
(30sxl:  d  =  .61;  10sx3:  d=  .68).  The  non-significant  10sx3  vs.  30sxl  differences  for  the 
initial  perfonnance  on  neutral  RAT  and  the  incubation  effect  size  suggest  that  reaching 
an  impasse  or  not  during  the  initial  problem-solving  phrase  has  no  effect  on  insight,  a 
result  strengthened  by  the  finding  of  increased  solution  of  misleading  problems  by  the 
10sx3  groups. 


We  revealed  a  lack  of  impact  of  impasse  on  insight.  We  suggest  that  the  role  of 
the  initial  problem-solving  attempt  is  for  successful  encoding,  rather  than  experiencing 
failure.  This  enables  individuals  to  maintain  the  task  information  in  some  non-immediate 
memory,  providing  the  basis  for  the  subsequent  unconscious  problem-solving  processing. 

STUDY  6:  Meta-Analysis  of  design  studies  on  example  impact  (Sio,  Kotovsky 
and  Cagan,  2015) 

A  meta-analytical  review  of  design  studies  (N=  43)  was  conducted  to  examine 
the  effects  of  examples  on  design  processes,  an  issue  that  has  been  the  focus  of  a  good  bit 
of  attention.  The  analysis  revealed  that  providing  example  solutions  made  individuals 
generate  more  example-related  ideas  and  produced  fewer  categories  of  ideas,  however, 
the  ideas  they  generated  were  more  novel.  There  was  also  a  positive  link  between  the 
degree  of  copying  and  the  quality  of  solution  ideas.  The  facilitatory  effects  on  novelty 
and  quality  were  stronger  when  fewer  and  less  common  examples  were  presented. 
Presenting  a  single  and  uncommon  example  may  encourage  individuals  to  shift  from 
traversing  between  different  parts  of  the  problem  space  to  conducting  a  deeper  search  in  a 
specific  and  remote  domain,  facilitating  the  generation  of  high-quality  and  novel  ideas. 

This  meta-analysis  examined  the  effects  of  positive  examples  on  design 
processes.  The  results  indicate  that  individuals  tend  to  copy  ideas  or  parts  of  ideas  from 
the  examples  and  produced  less  variety  of  ideas,  as  compared  to  those  who  did  not 
receive  any  example  solution  while  solving  the  design  task.  The  more  individuals  copy 
from  the  examples,  the  smaller  number  of  ideas  they  can  generate.  However,  looking  at 
examples  can  significantly  improve  the  quality  and  novelty  of  the  solution  ideas 
produced.  These  findings  are  consistent  with  our  prediction  that  the  presence  of 
examples  can  modify  the  search  strategy  from  a  broad  one  to  a  focused  one.  Although 
this  narrows  the  scope  of  search,  it  allows  a  more  in-depth  exploration,  and  in  turn, 
improves  solution  quality. 

Our  analyses  also  reported  a  significant  “timing”  effect;  presenting  examples  at 
the  beginning  of  problem  solving  produced  a  larger  positive  impact  on  design  solutions, 
as  compared  to  presenting  examples  during  problem  solving.  These  results  seem  to  be 
consistent  with  the  prediction  based  on  the  sunk  cost  effect  that  after  a  period  of  initial 


work  on  the  task,  individuals  have  developed  a  fixation  or  commitment  to  their  own 
approach;  and  this  should  lower  their  tendency  to  make  use  of  the  external  infonnation  to 
improve  the  current  problem  solving  approach. 

Our  analysis  did  not  uncover  a  strong  correlation  between  the  quantity  and  quality 
of  the  solution  ideas  (r  =  .007,  p  >.20).  This  challenges  the  common  assumption  among 
studies  of  brainstonning  that  the  greater  the  number  of  ideas  generated,  the  greater  the 
chance  of  producing  a  good  solution  (Osbom,  1963).  According  to  our  meta-analysis, 
generating  high  quantity  and  generating  high  quality  of  ideas  required  different  search 
strategies.  Looking  at  examples  led  individuals  to  explore  the  problem  space  more 
narrowly  and  deeply,  and  this  promoted  the  quality  and  novelty  of  the  design  solution. 
These  findings  support  the  use  of  examples  in  the  design  process  because,  in  real-life 
problem  solving,  people  usually  value  quality  over  quantity  of  solutions.  Providing  a 
single  rather  than  multiple  examples  allows  individuals  to  focus  on  fewer  domains  and  to 
search  deeper.  An  uncommon  example  will  direct  individuals  to  search  in  a  less  typical 
domain.  Together,  a  focused  search  in  an  uncommon  domain  should  facilitate  novel 
conceptual  combination.  These  support  our  prediction  that  looking  at  examples  can 
impact  how  individuals  explore  the  problem  space,  and  this  can  potentially  facilitate  the 
design  processes,  depending  on  the  number,  commonality,  and  the  timing  of  the 
presentation. 

STUDY  7:  An  Empirical  Study  of  Arrow’s  Impossibility  Theorem  (McComb, 
Goucher-Lambert,  and  Cagan,  2015) 

The  design  process  often  requires  work  by  teams,  rather  than  individuals.  During 
team-based  design  it  is  likely  that  situations  will  arise  in  which  the  members  of  the  team 
have  different  opinions,  yet  a  group  decision  must  still  be  made.  Unfortunately,  Arrow’s 
Impossibility  Theorem  indicates  that  there  is  no  method  for  aggregating  group 
preferences  that  will  always  satisfy  a  small  number  of  “fair”  conditions.  This  work  seeks 
to  identify  methods  of  combining  individual  preferences  that  can  come  close  to  satisfying 
Arrow’s  conditions.  Experiential  conjoint  analysis  was  used  to  obtain  empirical  utility 


functions  for  drinking  mug  designs.  A  number  of  functions  for  constructing  group 
preference  were  then  analyzed  using  both  randomly  generated  preferences  and 
preferences  derived  from  an  experiential  conjoint  survey.  The  analysis  involved  checking 
each  of  Arrow’s  conditions,  as  well  as  computing  the  likelihood  that  a  method  will  be 
susceptible  to  manipulation  by  a  dishonest  individual. 

This  work  took  an  empirical  approach  to  examine  several  methods  for  combining 
individual  preferences  into  a  group  preference.  Each  of  these  methods,  referred  to  in  this 
work  as  aggregation  functions,  was  analysed  in  terms  of  strategy-proofness  and 
Conditional  Arrow  Fairness.  Of  the  aggregation  functions  explored  in  this  work,  the 
Copeland  function  offers  the  highest  probability  of  Conditional  Arrow  Fairness  and  the 
highest  probability  of  strategy-proofness.  This  indicates  that  it  is  likely  to  return  a  fair 
result,  and  that  individuals  would  thus  have  no  incentive  to  provide  anything  but  their 
true  preference  for  the  alternatives.  This  result  is  true  for  both  empirical  preference 
profiles  and  randomly  generated  preference  profiles  (which  offer  a  worst-case  scenario 
for  forming  a  group  preference).  The  Copeland  function  could  be  applied  to  a  variety  of 
domains,  including  the  aggregation  of  preferences  from  user  surveys  and  decision¬ 
making  during  the  design  process. 

HIGHLIGHTS  OF  RESULTS 

The  studies  to  date,  while  covering  a  wide  variety  of  issues  and  using  both 
computational  modelling  and  human  experimental  investigations,  have  provided  some 
converging  findings  about  factors  that  detennine  the  efficacy  of  team  problem  solving. 
These  include  the  following: 


1 .  The  structure  of  teams  wherein  heterogeneous  teams  produced  better  outcomes 
and  were  better  able  to  respond  to  changes  in  the  problem  being  worked  on.  This  finding 
emerged  from  both  the  human  and  computational  studies  and  to  some  extent  was  based 
on  the  minimization  of  self  bias  among  team  members. 

2.  Despite  the  above  findings  about  the  advantages  of  heterogeneous  teams, 
which  suggests  variability  as  a  desirable  feature  of  problem  solving,  the  findings  from 
both  human  and  computational  modelling  show  that  the  more  efficacious  teams  explore 
much  more  efficiently  than  less  efficacious  teams.  They  do  search  broadly  briefly  but 
then  narrow  their  search  of  and  focus  more  deeply  in  the  problem  space  (albeit  in  better 
or  more  propitious  directions,  similar  to  expert  problem  solvers  in  other  domains.) 

Further  they  are  better  able  to  recover  from  sudden  changes  in  the  problems  that  occur  as 
they  work  on  them.  This  too  was  found  in  both  human  studies  and  our  computational 
models. 

3.  Presenting  problem  examples  to  solvers  was  found,  via  a  meta-analysis,  to  lead 
to  more  focused  search  (as  in  points  one  and  two  above)  and  to  the  recognition  that  the 
brainstorming  idea  that  quantity  of  ideas  always  produces  higher  quality  is  not  accurate. 
As  in  point  two,  narrower  more  focused  search  produced  better  outcomes  and  this  was 
particularly  true  in  the  case  of  the  presentation  of  unique  examples. 

4.  Finally,  the  ten  vs.  thirty  second  study  showed  that  reaching  an  impasse  is  not 
an  important  part  of  achieving  insights;  people  that  stopped  solving  and  switched  to 
another  problem  well  before  reaching  a  solution  or  an  impasse  do  better  overall  than 
those  allowed  to  steadily  work  on  one  problem  for  a  longer  time  and  then  the  other.  It 
also  showed  that  problems  can  be  carried  in  an  unconscious  store  while  working  on 


another  problem,  only  to  re-emerge  when  work  on  the  problem  is  re-activated,  again 
leading  to  more  focused  and  successful  search  for  a  solution.  These  and  other  of  our 
findings  suggest  intriguing  ways  that  team  perfonnance  and  problem  solving  search 
might  be  improved.  One  over-arching  outcome  of  this  work  is  the  finding  of  value  in 
rapidly  narrowing  or  converging  search,  often  after  a  brief  initial  period  of  broader 
search,  much  as  has  been  found  in  studies  of  expertise,  where  insightful  plunging  ahead 
on  a  propitious  pathway  leads  to  the  best  and  most  rapidly  acquired  solutions.  While 
much  remains  to  be  done,  the  findings  to  date  suggest  quite  strongly  that  team  structure, 
timing  of  problem-solving  work,  structure  of  communication  pathways  and  heterogeneity 
of  team  members’  background  information  all  can  converge  in  enhancing  the 
perfonnance  of  problem-solving  teams. 
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